System and method for three-dimensional augmented reality guidance for use of medical equipment

ABSTRACT

A medical guidance system providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a novice user of medical equipment having limited medical training, to achieve improved diagnostic or treatment outcomes.

BACKGROUND OF THE INVENTION

The present disclosure relates to systems for providing improvedtraining and guidance to equipment users, and more particularly systemsand methods for providing real-time, three-dimensional (3D) augmentedreality (AR) feedback-based guidance in the use of medical equipment bynovice users, to achieve improved diagnostic or treatment outcomes.

In many medical situations, diagnostic or treatment of medicalconditions, which may include life-saving care, must be provided bypersons without extensive medical training. This may occur becausetrained personnel are either not present or are unable to respond. Forexample, temporary treatment of broken bones occurring in remotewilderness areas must often be provided by a companion of the injuredpatient, or in some cases as self-treatment by the patient alone. Theneed for improved medical treatment in remote or extreme situations hasled to Wilderness First Aid training courses for hikers and backpackers.Battlefield injuries such as gunshot or blast injuries often requireimmediate treatment, e.g., within minutes or even seconds, by untrainedpersonnel under extreme conditions to stabilize the patient untiltransport is available. Injuries to maritime personnel may occur onsmaller vessels lacking a full-time physician or nurse, and illness orinjuries may require treatment by persons with little or no training.Similarly, injuries or illnesses occurring to persons in space (e.g.,the International Space Station) may also require treatment by personswith limited or incomplete medical training.

In many instances, such as maritime vessels and injuries in space,adequate medical equipment may be available, but the efficacy of the useof the equipment may be limited by the training level of thecaregiver(s). Improved treatment or diagnostic outcomes may be availableif improved training is available to caregivers having limited medicaltraining. As used herein, caregivers having little or no medicaltraining for the use of a particular medical device or medicaltechnology are referred to as “novice users” of the technology. Noviceusers may include persons having a rudimentary or working knowledge of amedical device or technology, but less than an expert or credentialedtechnician for such technology.

The present invention provides systems and methods for guiding medicalequipment users, including novice users. In some embodiments, systems ofthe present disclosure provide real-time guidance to a medical equipmentuser. In some embodiments, systems disclosed herein providethree-dimensional (3D) augmented-reality (AR) guidance to a medicaldevice user. In some embodiments, systems of the present disclosureprovide machine learning guidance to a medical device user. Guidancesystems disclosed herein may provide improved diagnostic or treatmentresults for novice users of medical devices. Use of systems of thepresent invention may assist novice users to achieve results comparableto those obtained by expert or credentialed medical caregivers for aparticular medical device or technology.

Although systems of the present invention may be described forparticular medical devices and medical device systems, persons of skillin the art having the benefit of the present disclosure will appreciatethat these systems may be used in connection with other medical devicesnot specifically noted herein. Further, it will also be appreciated thatsystems according to the present invention not involving medicalapplications are also within the scope of the present invention. Forexample, systems of the present invention may be used in many industrialor commercial settings to train users to operate may different kinds ofequipment, including heavy machinery as well as many types of precisioninstruments, tools, or devices. Accordingly, the particular embodimentsdisclosed above are illustrative only, as the invention may be modifiedand practiced in different but equivalent manners apparent to thoseskilled in the art having the benefit of the teachings herein. Examples,where provided, are all intended to be non-limiting. Furthermore,exemplary details of construction or design herein shown are notintended to limit or preclude other designs achieving the same function.The particular embodiments disclosed above may be altered or modifiedand all such variations are considered within the scope and spirit ofthe invention, which are limited only by the scope of the claims.

Many future manned spaceflight missions (e.g., by NASA, the EuropeanSpace Agency, or non-governmental entities) will require medicaldiagnosis and treatment capabilities that address the anticipated healthrisks and also perform well in austere, remote operational environments.Spaceflight-ready medical equipment or devices will need to be capableof an increased degree of autonomous operation, allowing the acquisitionof clinically relevant and diagnosable data by every astronaut, not justselect physician crew members credentialed in spaceflight medicine.

Augmented reality systems have been developed that provide step-by-stepinstructions to a user in performing a task. Such prior art systems mayprovide a virtual manual or virtual checklist for a particular task(e.g., performing a repair or maintenance procedure). In some systems,the checklist may be visible to the user via an augmented reality (AR)user interface such as a headset worn by the user. Providing the userwith step-by-step instructions or guidance may reduce the need fortraining for a wide variety of tasks, for example, by breaking a complextask into a series of simpler steps. In some instances,context-sensitive animations may be provided through an AR userinterface in the real-world workspace. Existing systems, however, may beunable to guide users in delicate or highly specific tasks that aretechnique-sensitive, such as many medical procedures or other equipmentrequiring a high degree of training for proficiency.

Thus, there is a need for AR systems capable of guiding a novice user ofequipment in real time through a wide range of unfamiliar tasks inremote environments such as space or remote wilderness (e.g., arctic)conditions. These may include daily checklist items (e.g., habitatsystems procedures and general equipment maintenance), assembly andtesting of complex electronics setups, and diagnostic and interventionalmedical procedures. AR guidance systems desirably would allow noviceusers to be capable of autonomously using medical and other equipment ordevices with a high degree of procedural competence, even where theoutcome is technique-sensitive.

SUMMARY

In one embodiment, the present invention comprises a medical guidancesystem (100) for providing real-time, three-dimensional (3D) augmentedreality (AR) feedback guidance in the use of a medical equipment system(200), the medical guidance system comprising: a medical equipmentinterface to a medical equipment system (200), wherein said medicalequipment interface is capable of receiving data from the medicalequipment system during a medical procedure performed by a user; anaugmented reality user interface (ARUI) (300) for presenting datapertaining to both real and virtual objects to the user during at leasta portion of the performance of the medical procedure; athree-dimensional guidance system (3DGS) (400) that is capable ofsensing real-time user positioning data relating to one or more of themovement, position, and orientation of at least a portion of the medicalequipment system (200) during said medical procedure performed by theuser; a library (500) containing 1) stored reference positioning datarelating to one or more of the movement, position, and orientation of atleast a portion of the medical equipment system (200) during a referencemedical procedure and 2) stored reference outcome data relating to anoutcome of said reference medical procedure; and a machine learningmodule (MLM) (600) for providing at least one of 1) position-based 3D ARfeedback to the user based on the sensed user positioning data and thereference positioning data, and 2) outcome-based 3D AR feedback to theuser based on data received from the medical equipment system during themedical procedure performed by the user and reference outcome data.

In one embodiment, the present invention comprises a medical guidancesystem (100) for providing real-time, three-dimensional (3D) augmentedreality (AR) feedback guidance in the use of a medical equipment system(200), the medical guidance system comprising: a computer 700 comprisinga medical equipment interface to a medical equipment system (200),wherein said medical equipment interface receives data from the medicalequipment system during a medical procedure performed by a user toachieve a medical procedure outcome; an AR interface to an AR headmounted display (HMD) for presenting information pertaining to both realand virtual objects to the user during the performance of the medicalprocedure; a guidance system interface (GSI) to a three-dimensionalguidance system (3DGS) (400) that senses real-time user positioning datarelating to one or more of the movement, position, and orientation of atleast a portion of the medical equipment system (200) within a volume ofa user's environment during a medical procedure performed by the user; alibrary (500) containing 1) stored reference positioning data relatingto one or more of the movement, position, and orientation of at least aportion of the medical equipment system (200) during a reference medicalprocedure and 2) stored reference outcome data relating to an outcome ofa reference performance of the reference medical procedure; and amachine learning module (MLM) (600) for providing at least one of 1)position-based 3D AR feedback to the user based on the sensed userpositioning data and 2) outcome-based 3D AR feedback to the user basedon the medical procedure outcome, the MLM (600) comprising aposition-based feedback module comprising a first module for receivingand analyzing real-time user positioning data; a second module forcomparing the user positioning data to the stored reference positioningdata, and a third module for generating real-time position-based 3D ARfeedback based on the output of the second module, and providing saidreal-time position-based 3D AR feedback to the user via the ARUI (300);and an outcome-based feedback module comprising a fourth module forreceiving real-time data from the medical equipment system (200) viasaid medical equipment interface as the user performs the medicalprocedure; a fifth module for comparing the real-time data received fromthe medical equipment system (200) as the user performs the medicalprocedure to the stored reference outcome data, and a sixth module forgenerating real-time outcome-based 3D AR feedback based on the output ofthe fifth module, and providing said real-time outcome-based 3D ARfeedback to the user via the ARUI (300).

In one embodiment, the present invention comprises a method forproviding real-time, three-dimensional (3D) augmented reality (AR)feedback guidance to a user of a medical equipment system, the methodcomprising: receiving data from a medical equipment system during amedical procedure performed by a user of the medical equipment toachieve a medical procedure outcome; sensing real-time user positioningdata relating to one or more of the movement, position, and orientationof at least a portion of the medical equipment system within a volume ofthe user's environment during the medical procedure performed by theuser; retrieving from a library at least one of 1) stored referencepositioning data relating to one or more of the movement, position, andorientation of at least a portion of the medical equipment system duringreference a medical procedure, and 2) stored reference outcome datarelating to a reference performance of the medical procedure; comparingat least one of 1) the sensed real-time user positioning data to theretrieved reference positioning data, and 2) the data received from themedical equipment system during a medical procedure performed by theuser to the retrieved reference outcome data; generating at least oneof 1) real-time position-based 3D AR feedback based on the comparison ofthe sensed real-time user positioning data to the retrieved referencepositioning data, and 2) real-time output-based 3D AR feedback based onthe comparison of the data received from the medical equipment systemduring a medical procedure performed by the user to the retrievedreference outcome data; and providing at least one of the real-timeposition-based 3D AR feedback and the real-time output-based 3D ARfeedback to the user via an augmented reality user interface (ARUI).

In one embodiment, the present invention comprises a method fordeveloping a machine learning model of a neural network for classifyingimages for a medical procedure using an ultrasound system, the methodcomprising: A) performing a first medical procedure using an ultrasoundsystem; B) automatically capturing a plurality of ultrasound imagesduring the performance of the first medical procedure, wherein each ofthe plurality of ultrasound images is captured at a defined samplingrate according to defined image capture criteria; C) providing aplurality of feature modules, wherein each feature module defines afeature which may be present in an image captured during the medicalprocedure; D) automatically analyzing each image using the plurality offeature modules; E) automatically determining, for each image, whetheror not each of the plurality of features is present in the image, basedon the analysis of each imagine using the feature modules; F)automatically labeling each image as belonging to one class of aplurality of image classes associated with the medical procedure; G)automatically splitting the plurality of images into a training set ofimages and a validation set of images; H) providing a deep machinelearning (DML) platform having a neural network to be trained loadedthereon, the DML platform having a plurality of adjustable parametersfor controlling the outcome of a training process; I) feeding thetraining set of images into the DML platform; J) performing the trainingprocess for the neural network to generate a machine learning model ofthe neural network; K) obtaining training process metrics of the abilityof the generated machine learning model to classify images during thetraining process, wherein the training process metrics comprise at leastone of a loss metric, an accuracy metric, and an error metric for thetraining process; L) determining whether each of the at least onetraining process metrics is within an acceptable threshold for eachtraining process metric; M) if one or more of the training processmetrics are not within an acceptable threshold, adjusting one or more ofthe plurality of adjustable DML parameters and repeating steps J, K, andL; N) if each of the training process metrics is within an acceptablethreshold for each metric, performing a validation process using thevalidation set of images; O) obtaining validation process metrics of theability of the generated machine learning model to classify imagesduring the validation process, wherein the validation process metricscomprise at least one of a loss metric, an accuracy metric, and an errormetric for the validation process; P) determining whether each of thevalidation process metrics is within an acceptable threshold for eachvalidation process metric; Q) if one or more of the validation processmetrics are not within an acceptable threshold, adjusting one or more ofthe plurality of adjustable DML parameters and repeating steps J-P; andR) if each of the validation process metrics is within an acceptablethreshold for each metric, storing the machine learning model for theneural network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram view of a system for providing real-time,three-dimensional (3D) augmented reality (AR) guidance in the use of amedical device system.

FIG. 2 is a diagram showing communication among the modules of areal-time, 3D AR feedback guidance system for the use of an ultrasoundsystem, according to one embodiment.

FIG. 3 is a diagram showing an ultrasound system that may includemultiple modes of operation, involving different levels of AugmentedReality functions.

FIG. 4 is a diagram illustrating major software components in anexperimental architecture for a system according to one embodiment ofthe present disclosure.

FIG. 5 is a software component diagram with more details of the softwarearchitecture of FIG. 4.

FIG. 6 is a flowchart of a method for developing a machine learningmodule using manually prepared data sets.

FIG. 7 is a block diagram of a machine learning development module.

FIG. 8 is a flowchart of a method for developing a machine learningmodule using automatically prepared data sets.

FIGS. 9A-9F are ultrasound images that illustrate one or more featuresthat may be used to classify ultrasound images.

FIGS. 10A and 10B are ultrasound images illustrating isolating orlabeling specific structures in each image.

DESCRIPTION

Exemplary embodiments are illustrated in referenced figures of thedrawings. The embodiments disclosed herein are considered illustrativerather than restrictive. No limitation on the scope of the technologyand on the claims that follow is to be imputed to the examples shown inthe drawings and discussed here.

As used herein, the term “augmented reality” refers to display systemsor devices capable of allowing a user to sense (e.g., visualize) objectsin reality (e.g., a patient on an examination table and a portion of amedical device used to examine the patient), as well as objects that arenot present in reality but which relate in some way to objects inreality, but which are displayed or otherwise provided in a sensorymanner (e.g., visually or via sound) in the AR device. Augmented realityas used herein is a live view of a physical, real-world environment thatis augmented to a user by computer-generated perceptual information thatmay include visual, auditory, haptic (or tactile), somatosensory, orolfactory components. The augmented perceptual information is overlaidonto the physical environment in spatial registration so as to beperceived as immersed in the real world. Thus, for example, augmentedvisual information is displayed relative to one or more physical objectsin the real world, and augmented sounds are perceived as coming from aparticular source or area of the real world. This could include, asnonlimiting examples, visual distance markers between particular realobjects in the AR display, or grid lines allowing the user to gaugedepth and contour in the visual space, and sounds, odors, and tactileinputs highlighting or relating to real objects.

A well-known example of AR devices are heads-up displays on militaryaircraft and some automobiles, which allow the pilot or driver toperceive elements in reality (the landscape and/or aerial environment)as well as information related to the environment (e.g., virtual horizonand plane attitude/angle, markers for the position of other aircraft ortargets, etc.) that is not present in reality but which is overlaid onthe real environment. The term “augmented reality” (AR) is intended todistinguish systems herein from “virtual reality” (VR) systems thatdisplay only items that are not actually present in the user's field ofview. Examples of virtual reality systems include VR goggles for gamingthat present information to the viewer while blocking entirely theviewer's perception of the immediate surroundings, as well as thedisplay on a television screen of the well-known “line of scrimmage” and“first down” markers in football games. While the football fieldactually exists, it is not in front of the viewer; both the field andthe markers are only presented to the viewer on the television screen.

In one aspect of the present disclosure, a 3D AR system according to thepresent disclosure may be provided to a novice medical device user forreal-time, three-dimensional guidance in the use of an ultrasoundsystem. Ultrasound is a well-known medical diagnostic and treatmenttechnology currently used on the International Space Station (ISS) andplanned for use in future deep-space missions. A variety of ultrasoundsystems may be used in embodiments herein. In one nonlimiting example,the ultrasound system by be the Flexible Ultrasound System (FUS), anultrasound platform being developed by NASA and research partners foruse in space operations.

FIG. 1 is a block diagram view of one embodiment of a system forproviding real-time, three-dimensional (3D) augmented reality (AR)guidance in the use of medical equipment by novice users having limitedmedical training, to achieve improved diagnostic or treatment outcomes.The system includes a computer 700 in communication with additionalsystem components. Although FIG. 1 is a simplified illustration of oneembodiment of a 3D AR guidance system 100, computer 700 includes variousinterfaces (not shown) to facilitate the transfer and receipt ofcommands and data with the other system components. The interfaces incomputer 700 may comprise software, firmware, hardware or combinationsthereof.

In one embodiment, computer 700 interfaces with a medical equipmentsystem 200, which in one embodiment may be an ultrasound system. Inother embodiments, different medical equipment, devices or systems maybe used instead of or in addition to ultrasound systems. In theembodiment depicted in FIG. 1, the medical equipment system 200 isincluded as part of the 3D AR guidance system 100. In one embodiment,the medical equipment system 200 is not part of the guidance system 100;instead, guidance system 100 includes a medical equipment systeminterface (MESI) to communicate with the medical equipment system 200,which may comprise any of a variety of available medical device systemsin a “plug-and-play” manner.

In one embodiment, the 3D AR guidance system 100 also includes anaugmented reality user interface (ARUI) 300. The ARUI 300 may comprise avisor having a viewing element (e.g., a viewscreen, viewing shield orviewing glasses) that is partially transparent to allow a medicalequipment user to visualize a workspace (e.g., an examination room,table or portion thereof). In one embodiment, the ARUI 300 includes ascreen upon which virtual objects or information can be displayed to aida medical equipment user in real-time (i.e., with minimal delay betweenthe action of a novice user and the AR feedback to the action,preferably less than 2 seconds, more preferably less than 1 second, mostpreferably 100 milliseconds or less). As used herein, three-dimensional(3D) AR feedback refers to augmented reality sensory information (e.g.,visual or auditory information) providing to the user based at least inpart on the actions of the user, and which is in spatial registrationwith real world objects perceptible (e.g., observable) to the user. TheARUI 300 provides the user with the capability of seeing all or portionsof both real space and virtual information overlaid on or inregistration with real objects visible through the viewing element. TheARUI 300 overlays or displays (and otherwise presents, e.g., as soundsor tactile signals) the virtual information to the medical equipmentuser in real time. In one embodiment, system also includes an ARUIinterface (not shown) to facilitate communication between the headsetand the computer 700. The interface may be located in computer 700 orARUI 300, and may comprise software, firmware, hardware, or combinationsthereof

A number of commercially available AR headsets may be used inembodiments of the present invention. The ARUI 300 may include one ofthese commercially available headsets. In the embodiment depicted inFIG. 1, the ARUI is included as part of the 3D AR guidance system 100.In an alternative embodiment, the ARUI 300 is not part of the guidancesystem 100, and guidance system 100 instead includes an ARUI interface,which may be provided as software, firmware, hardware or a combinationthereof in computer 700. In this alternative embodiment, the ARUIinterface communicates with the ARUI 300 and one or more other systemcomponents (e.g., computer 700), and ARUI 300 may comprise any ofabove-described commercially available headsets in a “plug-and-play”manner.

The embodiment of FIG. 1 further comprises a three-dimensional guidancesystem (3DGS) 400 that senses or measures real objects in real-timewithin a volume in the user's environment. The 3DGS 400 is used to mapvirtual information onto the real objects for display or other sensorypresentation to the user via the ARUI 300. Although a variety ofdifferent kinds of three-dimensional guidance systems may be used invarious embodiments, all such systems 400 determine the position of oneor more objects, such as a moveable sensor, relative to a fixedtransmitter within a defined operating volume. The 3DGS 400 additionallyprovides the positional data to one or more other modules in FIG. 1(e.g., to the machine learning module 600) via computer 700.

In one embodiment, the 3DGS 400 senses real-time user positioning datawhile a novice user performs a medical procedure. User positioning datarelates to or describes one or more of the movement, position, andorientation of at least a portion of the medical equipment system 200while the user (e.g., a novice) of performs a medical procedure. Userpositioning data may, for example, include data defining the movement ofan ultrasound probe during an ultrasound procedure performed by theuser. User positioning data may be distinguished from user outcome data,which may be generated by medical equipment system 200 while the userperforms a medical procedure, and which includes data or informationindicating or pertaining to the outcome of a medical procedure performedby the user. User outcome data may include, as a nonlimiting example, aseries of ultrasound images captured while the user performs anultrasound procedure, or an auditory or graphical record of a patient'scardiac activity, respiratory activity, brain activity, etc.

In one embodiment, the 3DGS 400 is a magnetic GPS system such as VolNav,developed by GE, or other magnetic GPS system. Magnetic GPS trackingsystems While magnetic GPS provides a robust, commercially availablemeans of obtaining precision positional data in real-time, in someenvironments (e.g., the International Space Station) magnetic GPS may beunable to tolerate the small magnetic fields prevalent in suchenvironments. Accordingly, in some embodiments, alternative oradditional 3D guidance systems for determining the position of thepatient, tracking the user's actions, or tracking one or more portionsof the medical equipment system 200 (e.g., an ultrasound probe) may beused instead of a magnetic GPS system. These may include, withoutlimitation, digital (optical) camera systems such as the DMA6SA andOptitrack systems, infrared cameras, and accelerometers and/orgyroscopes.

In the case of RGB (color) optical cameras and IR (infrared) depthcamera systems, the position and rotation of the patient, the user'sactions, and one or more portions of the medical equipment system may betracked using non-invasive external passive visual markers or externalactive markers (i.e., a marker emitting or receiving a sensing signal)coupled to one or more of the patient, the user's hands, or portions ofthe medical equipment. The position and rotation of passive markers inthe real world may be measured by the depth cameras in relation to avolume within the user's environment (e.g., an operating room volume),which may be captured by both the depth cameras and color cameras.

In the case of accelerometers and gyroscopes, the combination ofacceleration and gyroscopes comprises inertial measurement units (IMUs),which can measure the motion of subjects in relation to a determinedpoint of origin or reference plane, thereby allowing the position androtation of subjects to be derived. In the case of a combination ofcolor cameras, depth cameras, and IMUs, the aggregation of measuredposition and rotation data (collectively known as pose data) becomesmore accurate.

In an alternative embodiment, the 3DGS 400 is not part of the guidancesystem 100, and guidance system 100 instead includes a 3DGS interface,which may be provided as software, firmware, hardware or a combinationthereof in computer 700. In this alternative embodiment, the 3DGSinterface communicates with the 3DGS 400 and one or more other systemcomponents (e.g., computer 700), and 3DGS 400 interfaces with the system100 (e.g., via computer 700) in a “plug-and-play” manner.

In one embodiment of the invention, the 3DGS 400 tracks the user'smovement of an ultrasound probe (provided as part of medical equipmentsystem 200) relative to the body of the patient in a defined examinationarea or room. The path and position or orientation of the probe may becompared to a desired reference path and position/orientation (e.g.,that of an expert user such as a physician or ultrasound technicianduring the examination of a particular or idealized patient forvisualizing a specific body structure). This may include, for example,an examination path of an expert user for longitudinal orcross-sectional visualization of a carotid artery of a patient using theultrasound probe.

Differences between the path and/or position/orientation of the probeduring an examination performed by a novice user in real-time, and anidealized reference path or position/orientation (e.g., as taken duringthe same examination performed by an expert), may be used to providereal-time 3D AR feedback to the novice user via the ARUI 300. Thisfeedback enables the novice user to correct mistakes or incorrect usageof the medical equipment and achieve an outcome similar to that of theexpert user. The real-time 3D AR feedback may include visual information(e.g., a visual display of a desired path for the novice user to takewith the probe, a change in the position or orientation of the probe,etc.), tactile information (e.g., vibrations or pulses when the noviceuser is in the correct or incorrect position), or sound (e.g., beepingwhen the novice user is in the correct or incorrect position).

Referring again to FIG. 1, system 100 further includes a library 500 ofinformation relating to the use of the medical equipment system 200. Thelibrary 500 includes detailed information on the medical equipmentsystem 200, which may include instructions (written, auditory, and/orvisually) for performing one or more medical procedures using themedical equipment system, and reference information or data in the useof the system to enable a novice user to achieve optimal outcomes (i.e.,similar to those of an expert user) for those procedures. In oneembodiment, library 500 includes stored reference information relatingto a reference performance (e.g., an expert performance) of one or moremedical procedures. This may include one or both of stored referencepositioning data, which relates to or describes one or more of themovement, position, and orientation of at least a portion of the medicalequipment system 200 during a reference performance of a medicalprocedure, and stored reference outcome data, which includes data orinformation indicating or pertaining to a reference outcome of a medicalprocedure (e.g., when performed by an expert). Reference positioningdata may include, as a nonlimiting example, data defining the referencemovement of an ultrasound probe during a reference performanceperforming an ultrasound procedure. Reference outcome data may include,as a nonlimiting example, data comprising part or all of the outcome ofa medical procedure, such as a series of ultrasound images capturing oneor more desired target structures of a patient's body, or an auditory orgraphical record of a patient's cardiac activity, respiratory activity,brain activity, etc. In some embodiments, the library 500 may includepatient data, which may be either generic data relating to the use ofthe medical equipment system on a number of different patients, orpatient-specific data (i.e., data relating to the use of the equipmentsystem on one or more specific patients) to guide a user of the medicaldevice to treat a specific patient. Additional information (e.g., usermanuals, safety information, etc.) for the medical equipment system 200may also be present in the library 500.

A machine learning module (MLM) 600 is provided to generate feedback toa novice user of the system 100, which may be displayed in the ARUI 300.MLM 600 is capable of comparing data of a novice user's performance of aprocedure or task to that of a reference performance (e.g., by an expertuser). MLM 600 may receive real-time data relating to one or both of 1)the movement, position or orientation (“positioning data”) of a portionof the medical equipment 200 during the novice user's performance of adesired medical task (e.g., the motion, position and orientation of anultrasound probe as manipulated by a novice user to examine a patient'scarotid artery), and 2) data received from the medical equipment 200relating to an outcome of the medical procedure (“outcome data”).

As previously noted, the positioning data (e.g., relating to thereal-time motion, position or orientation an ultrasound probe during useby a novice user) is obtained by the 3DGS 400, which senses the positionand/or orientation of a portion of the medical device at a desiredsampling rate (e.g., 100 times per second (Hz) up to 0.1 Hz or onceevery 10 seconds). The positioning data is then processed by one or moreof the 3DGS 400, computer 700, or MLM 600 to determine the motion andposition/orientation of a portion of the medical equipment system 200 asmanipulated by the novice user during the medical procedure.

The MLM 600 includes a plurality of modules, which may comprisesoftware, firmware or hardware, for generating and providing one or bothof position-based and outcome-based feedback to user. In one embodiment,MLM 600 includes a first module for receiving and processing real-timeuser positioning data, a second module for comparing the real-time userpositioning data (obtained by the 3DGS 400) to corresponding storedreference positioning data in patient library 500 of the motion andposition/orientation obtained during a reference performance of the samemedical procedure or task. Based on the comparison of the movements ofthe novice user and the reference performance, the MLM 600 may thendetermine discrepancies or variances of the performance of the noviceuser and the reference performance. A third module in the MLM generatesreal-time position-based 3D AR feedback based on the comparisonperformed by the second module, and provides the real-timeposition-based 3D AR feedback to the user via the ARUI 300. Thereal-time, 3D AR position-based feedback may include, for example,virtual prompts to the novice user to correct or improve the novice'suser's physical performance (i.e., manipulation of the relevant portionof the medical equipment system 200) of the medical procedure or task.The feedback may include virtual still images, virtual video images,sounds, or tactile information. For example, the MLM 600 may cause theARUI 300 to display a virtual image or video instructing the novice userto change the orientation of a probe to match a desired reference (e.g.,expert) orientation, or may display a correct motion path to be taken bythe novice user in repeating a prior reference motion, with color-codingto indicate portions of the novice user's prior path that were erroneousor sub-optimal. In some embodiments, the MLM 600 may cause the ARUI 300to display only portions of the novice user's motion that must becorrected.

In one embodiment, the MLM 600 also includes a fourth module thatreceives real-time data from the medical equipment system 200 itself(e.g., via an interface with computer 700) during a medical procedureperformed by the novice user, and a fifth module that compares that datato stored reference outcome data from library 500. For example, the MLM600 may receive image data from an ultrasound machine during use by anovice user at a specified sampling rate (e.g., from 100 Hz to 0.1 Hz),or specific images captured manually by the novice user, and may comparethe novice user image data to stored reference image data in library 500obtained during a reference performance of the medical procedure (e.g.,by an expert user such as an ultrasound technician).

The MLM 600 further includes a sixth module that generates real-timeoutcome-based feedback based on the comparison performed in the fifthmodule, and provides real-time, 3D AR outcome-based feedback to the uservia the ARUI 300. The real-time outcome-based feedback may includevirtual prompts to the user different from, or in addition to, thevirtual prompts provided from the positioning data. Accordingly, theoutcome data provided by MLM 600 may enable the novice user to furtherrefine his or her use of the medical device, even when the positioningcomparison discussed above indicates that the motion, position and/ororientation of the portion of the medical device manipulated by thenovice user is correct. For example, the MLM 600 may use the outcomedata from the medical device 200 and library 500 to cause the ARUI 300to provide a virtual prompt instructing the novice user to press anultrasound probe deeper or shallower into the tissue to the focus theultrasound image on a desired target such as a carotid artery. Thevirtual prompt may comprise, for example, an auditory instruction or avisual prompt indicating the direction in which the novice user shouldmove the ultrasound probe. The MLM 600 may also indicate to the noviceuser whether an acceptable and/or optimal outcome in the use of thedevice has been achieved.

It will be appreciated from the foregoing that MLM 600 can generate andcause ARUI 300 to provide virtual guidance based on two different typesof feedback, including 1) position-based feedback based on thepositioning data from the 3DGS 400 and 2) outcome-based feedback basedon outcome data from the medical equipment system 200. In someembodiments the dual-feedback MLM 600 provides a tiered guidance to anovice user: the position-based feedback is used for high-level promptsto guide the novice user in performing the overall motion for a medicalprocedure, while the outcome-based feedback from the medical device 200may provide more specific guidance for fine or small movements inperforming the procedure. Thus, MLM 600 may in some instances provideboth “coarse” and “fine” feedback to the novice user to help achieve aprocedural outcome similar to that of a reference outcome (e.g.,obtained from an expert user). Additional details of the architectureand operation of the MLM is provided in connection with subsequentfigures.

Referring again to FIG. 1, software interfaces between the variouscomponents of the system 100 are included to allow the system components200, 300, etc. to function together. A computer 700 is provided thatincludes the software interfaces as well as various other computerfunctionalities (e.g., computational elements, memory, processors,input/output elements, timers, etc.).

FIG. 4 illustrates the major software components in an experimentalarchitecture for a system according to FIG. 1 for providing real-time 3DAR guidance in the use of a Flexible Ultrasound System (FUS) developedby NASA with a Microsoft HoloLens Head Mounted Display ARUI. Inparticular, FIG. 4 illustrates a software architecture for oneembodiment of interfaces between computer 700 and 1) a medical equipmentsystem 200 (i.e., the Flexible Ultrasound System), and 2) an ARUI 300(i.e., the HoloLens Head Mounted Display ARUI). In some embodiments,these interfaces may be located within the medical equipment system orthe ARUI, respectively, rather than in a separate computer.

Software components 402-410 are the software infrastructure modules usedto integrate the FUS Research Application (FUSRA) 430 with the HoloLensHead Mounted Display (HMD) augmented reality (AR) application module412. Although a wide range of architectures are possible, theintegration for the experimental system of FIG. 4 uses a message queuingsystem for communication of status information, as well as command andstate information (3D spatial data and image frame classification bydeep machine learning) between the HoloLens ARUI and the FUS.Separately, the FUS ultrasound images are provided by a web server(discussed more fully below) dedicated to providing images for theHoloLens HMD AR application module 412 as an image stream.

The HoloLens HMD AR application module 412 software components arenumbered 412-428. The main user interfaces provided by the HoloLens HMDAR application 412 are a Holograms module 414 and a Procedure Managermodule 416. The Holograms module 414 blends ultrasound images, realworld objects and 3D models, images and graphical clues for display inthe HMD HoloLens ARUI. The Procedure Manager module 416 provides statusand state for the electronic medical procedure being performed.

The FUS Research Application (FUSRA) module 430 components are numbered430-440. The FUSRA module 430 will have capability to control the FUSultrasound scan settings when messages (commands) are received by thecomputer from the FUS to change scan settings. Specific probe andspecific scan settings are needed for specific ultrasound procedures.One specific example is the gain scan setting for the ultrasound, whichis controlled by the Processing Control Dialog module 434 using theMessage Queue 408 and C++ SDK Processing Chain 446 to control scansettings using C++ FUS shared memory (FIG. 5).

The FUSRA module 430 will have the capability to provide FUS ultrasoundimages in near-real time (high frame rate per second) so the HoloLensHead Mounted Display (HMD) Augmented Reality (AR) application module 412can display the image stream. The FUSRA module 430 provides JPEG imagesas MJPEG through a web server 438 that has been optimized to display animage stream to clients (e.g., HoloLens HMD AR application module 412).The Frame Output File 436 (and SDL JPEG Image from FUS GPU, FIG. 5)provide images for the Paparazzo Image Web Server 406 and Image WebServer 438.

The FUSRA module 430 is also capable of providing motion tracking 3Dcoordinates and spatial awareness whenever the 3D Guidance System (3DGS)400 (FIG. 1) is operating and providing data. The FUSRA module 430 usesthe positional data received from the 3DGS 400 for motion tracking. The3DGS 400 will provide spatial data (e.g., 3D position and rotation data)of tracked objects (e.g., the ultrasound probe) to clients using aMessage Queue module 408. This is also referenced in FIG. 4 by 3DGController 420 and Message Queue module 402, which communicates with the3DGS 400 of FIG. 1.

The FUS software development kit (SDK) in the FUSRA module 430 containsrudimentary image processing software to provide JPEG images to theFUSRA. The FUSRA module 430 contains additional image processing formonitoring and improving image quality, which is part of the C++ FUS SDKFramework 450 providing images to the Image Web Server 438 in FIG. 4.

The FUSRA module 430 uses the machine learning module (MLM) 600 (FIG. 1)for providing deep machine learning capabilities. The MLM 600 includes aneural network to be “trained” so that it “learns” how to interpretultrasound images obtained by a novice user to compare to a “baseline”set of images from a reference performance of an ultrasound procedure(e.g., by an expert). The MLM 600 will generate image classificationdata to classify ultrasound images. The classification of images is thebasis for the real-time outcome-based guidance provided to the noviceuser via the ARUI 300 (e.g., HoloLens Head Mounted Display device)during the performance of an ultrasound procedure. The imageclassification data will be provided to the HoloLens HMD AR applicationmodule 412 through a message queue 410 using the Computational Networktoolkit (CNTK) 454 in FIG. 4.

The HoloLens HMD AR application module 412 provides a hands-free headmounted display ARUI platform for receiving and viewing real-timefeedback during an ultrasound procedure. It also allows the novice userto focus on the patient without having to focus away from the patientfor guidance.

The HoloLens HMD AR application module uses the HoloLens HMD platformfrom Microsoft and the Unity 3D game engine 442 from Unity. The HoloLensHMD AR application module 412 displays guidance during execution of theultrasound medical procedure with AR visual clues and guidance, inaddition to the ultrasound image that is also visible through theHoloLens HMD display. The HoloLens HMD AR application module 412 alsohas the capability to control the FUS scan settings as part of theprocedure setup.

The architecture is designed to be extended to utilize electronicprocedures or eProc. Once an electronic procedure is created (using anelectronic procedure authoring tool) the procedure can be executed withthe Procedure Manager module 416.

The HoloLens HMD AR application module 412 includes the capability toalign 3D models and images in the holographic scene with real worldobjects like the ultrasound unit, its probe and the patient. Thisalignment allows virtual models and images to align with real worldobjects for rendering in the HoloLens head mounted display.

The HoloLens HMD AR application module 412 uses voice-based navigationby the novice user to maintain hands free operation of the ultrasoundequipment, except during initialization when standard keyboard or otherinterfaces may be used for control. Voice command modules in FIG. 4include the User Interface Behaviors module 418, User Interface Layers422, and Scene Manager 424.

The HoloLens HMD AR application module 412 also is capable ofcontrolling the FUS settings as part of the procedure setup. Thisfunction is controlled by the 3DG 400 (FIG. 1) using the Message Queue402.

The HoloLens HMD AR application module 412 provides an Image Streammodule 404 for display of ultrasound images that can be overlaid withguidance clues prompting the user to correctly the position theultrasound probe. The HoloLens HMD AR application 412 is also capable ofdisplaying 3D models and images in the HoloLens HMD along with realworld objects like the ultrasound, its probe and the patient. TheHoloLens HMD display allows virtual models and images to render overreal world objects within the novice user's view. This is provided theImage Streamer 404 supplying images to the Holograms module 414 throughthe User Interface Layers module 422, User Interface Models module 426,and Scene Manager module 424. This image stream is the same kind ofimage as a regular display device but tailored for HMD.

FIG. 5 shows a software component diagram with more details of thesoftware architecture of FIG. 4. Specifically, it shows the componentsallocated to the FUSRA module 430 and to the HoloLens HMD AR applicationmodule 412. Interactions among the software components are denoted bydirectional arrows and labels in the diagram. The FUSRA module 430 andthe HoloLens HMD AR application module 412 use robust connectivity thatis light weight and performs well. This is depicted in Figure by usingedges components of FIG. 4, which include Message Queue modules 402,408, and 410, as well as Image Streamer module 404 and Paparazzo ImageWeb Server module 406. The latter is dedicated to supplying theultrasound image stream from the FUSRA module 430 to the HoloLens HMD ARapplication module 412. While the Paparazzo Image Web Server module 406in some embodiments also sends other data to the HoloLens HMD ARapplication module 412, in one embodiment it is dedicated to images.Message Queues 402, 408, 410 are used for FUS scan setting controls andvalues, motion tracking, image classification, and other state dataabout the FUS. In addition, they provide much of the data required forthe MLM 600 to generate and provide guidance to the HoloLens HMD ARapplication module 412. The architecture of FIGS. 4 and 5 isillustrative only and is not intended to be limiting.

An embodiment of a particular system for real-time, 3D AR feedbackguidance for novice users of an ultrasound system, showing communicationbetween the system modules, is provided in FIG. 2. An ultrasound system210 is provided for use by a novice user 50 to perform an ultrasoundmedical procedure on a patient 60. The ultrasound system 210 may be anyof a number of existing ultrasound systems, including the previouslydescribed Flexible Ultrasound System (FUS) for use in a spaceexploration environment. Other ultrasound systems, such as the GE LogiqE90 ultrasound system, and the Titan portable ultrasound system made bySonosite, may be used, although it will be appreciated that differentsoftware interfaces may be required for different ultrasound systems.

The ultrasound system 210 may be used by novice user 50 to perform avariety of diagnostic procedures for detecting one or more medicalconditions, which may include without limitation carotid assessments,deep vein thrombosis, cardiogenic shock, sudden cardiac arrest, andvenous or arterial cannulation. In addition to the foregoingcardiovascular uses, the ultrasound system 210 may be used to performprocedures in many other body systems, including body systems that mayundergo changes during zero gravity space operations. Procedures thatmay be performed include ocular examinations, musculoskeletalexaminations, renal evaluation, and cardiac (i.e., heart) examinations.

In some embodiments, imaging data from the ultrasound system 210 isdisplayed on an augmented reality user interface (ARUI) 300. A widevariety of available ARUI units 300, many comprising a Head-MountedDisplay (HMD), may be used in systems of the present invention. Thesemay include the Microsoft HoloLens, the Vuzix Wrap 920AR and Star 1200,Sony HMZ-T1, Google Glass, Oculus Rift DK1 and DK2, Samsung GearVR, andmany others. In some embodiments, the system can support multiple ARUIs300, enabling multiple or simultaneous users for some procedures ortasks, and in other embodiments allowing third parties to view theactions of the user in real time (e.g., suitable for allowing an expertuser to train multiple novice users).

Information on a variety of procedures that may be performed by noviceuser 50 may be provided by Library 500, which in some embodiments may bestored on a cloud-based server as shown in FIG. 2. In other embodiments,the information may be stored in a conventional memory storage unit. Inone embodiment, the library 500 may obtain and display via the ARUI 300an electronic medical procedure 530, which may include displayingstep-by-step written, visual, audio, and/or tactile instructions forperforming the procedure.

As shown in FIG. 2, a 3D guidance system (3DGS) 400 may map the spacefor the medical procedure and may track the movement of a portion of themedical device system 100 by a novice user (50) as he or she performs amedical procedure. In one nonlimiting example, the 3DGS 400 track themovement of the probe 215 of the ultrasound system 210, which is used toobtain images.

In some embodiments, the 3DGS 400, either alone or in combination withlibrary 500 and/or machine learning module (MLM) 600, may cause ARUI 300to display static markers or arrows to complement the instructionsprovided by the electronic medical procedure 530. The 3DGS 400 cancommunicate data relating to the movements of probe 215, while a user isperforming a medical procedure, to the MLM 600.

The machine learning module (MLM) 600 compares the performance of thenovice user 50 to that of a reference performance (e.g., by an expertuser) of the same procedure as the novice user. As discussed regardingFIG. 1, MLM 600 may provide real-time feedback to the novice user viathe ARUI 300. The real-time feedback may include either or both ofposition-based feedback using data from the 3DGS 400, as well asoutcome-based feedback from the ultrasound system 210.

The MLM 600 generates position-based feedback by comparing the actualmovements of a novice user 50 (e.g., using positioning data receivedfrom the 3DGS 400 tracking the movement of the ultrasound probe 215) toreference data for the same task. In one embodiment, the reference datais data obtained by an expert performing the same task as the noviceuser. The reference data may be either stored in MLM 600 or retrievedfrom library 500 via a computer (not shown). Data for a particularpatient's anatomy may also be stored in library 500 and used by the MLM600.

Based on the comparison of the novice user's movements to those of theexpert user, the MLM 600 may determine in real time whether the noviceuser 50 is acceptably performing the task or procedure (i.e., within adesired margin of error to that of an expert user). The MLM 600 maycommunicate with ARUI 300 to display real time position-based feedbackguidance in the form of data and/or instructions to confirm or correctthe user's performance of the task based on the novice user movementdata from the 3DGS 400 and the reference data. By generating feedback inreal-time as the novice user performs the medical procedure, MLM 600thereby enabling the novice user to correct errors or repeat movementsas necessary to achieve an outcome for the medical procedure that iswithin a desired margin to that of reference performance.

In addition to the position-based feedback generated from position datareceived from 3DGS 400, MLM 600 in the embodiment of FIG. 2 alsoprovides outcome-based feedback based on comparing the ultrasound imagesgenerated in real-time by the novice user 50 to reference images for thesame medical procedure stored in the library 500. Library 500 mayinclude data for multiple procedures and/or tasks to be performed usinga medical device system such as ultrasound system 210. In alternativeembodiments, only one type of real-time feedback (i.e., position-basedfeedback or outcome-based feedback) is provided to guide a novice user.The type of feedback (i.e., based on position or the outcome of themedical procedure) may be selected based on the needs of the particularlearning environment. In some types of equipment, for example, feedbackgenerated by MLM solely based on the novice user's manipulation of aportion of the equipment (i.e., movements of a probe, joystick, lever,rod, etc.) may be adequate to correct the novice user's errors, while inother systems information generated based on the outcome achieved by theuser (outcome-based feedback) may be adequate to correct the noviceuser's movements without position-based feedback.

Although FIG. 2 is directed to an ultrasound system, it will beappreciated that in systems involving different types of medical (e.g.,a cardiogram), or non-medical equipment, the outcome-based feedback maybe based not on the comparison of images but on numerical, graphical, orother forms of data. Regardless of the type of equipment used,outcome-based feedback is generated by the MLM 600 based on datagenerated by the equipment that indicates whether or not the novice usersuccessfully performed a desired task or procedure. It will be furtherappreciated that in some embodiments of the present invention,outcome-based feedback may be generated using a neural network, while inother embodiments, a neural network may be unnecessary.

In one embodiment, one or both of real-time motion-based feedback andoutcome-based feedback may be used to generate a visual simulation(e.g., as a narrated or unnarrated video displayed virtually to thenovice user in the ARUI 300 (e.g., a HoloLens headset). In this way, thenovice user may quickly (i.e., within seconds of performing a medicalprocedure) receive feedback indicating deficiencies in technique orresults, enabling the user to improve quickly and achieve outcomessimilar to those of a reference performance (e.g., an expertperformance) of the medical or other equipment.

In one embodiment, the novice user's performance may be tracked overtime to determine areas in which the novice user repeatedly fails toimplement previously provided feedback. In such cases, trainingexercises may be generated for the novice user focusing on the specificmotions or portions of the medical procedure that the novice user hasfailed to correct, to assist the novice user to achieve improvedresults. For example, if the novice user fails to properly adjust theangle of an ultrasound proper at a specific point in a medicalprocedure, the MLM 600 and/or computer 700 may generate a video fordisplay to the user that this limited to the portion of the procedurethat the user is performing incorrectly. This allows less time to bewasted having the user repeat portions of the procedure that the user iscorrectly performing, and enables the user to train specifically onareas of incorrect technique.

In another embodiment, the outcome-based feedback may be used to detectproduct malfunctions. For example, if the images being generated by anovice user at one or more points during a procedure fail to correspondto those of a reference (e.g., an expert), or in some embodiments by thenovice user during prior procedures, the absence of any other basis forthe incorrect outcome may indicate that the ultrasound machine ismalfunctioning in some way.

In one embodiment, the MLM 600 may provide further or additionalinstructions to the user in real-time by comparing the user's responseto a previous real-time feedback guidance instruction to refine orfurther correct the novice user's performance of the procedure. Byproviding repeated guidance instruction as the novice user refineshis/her technique, MLM 600 may further augment previously-providedinstructions as the user repeats a medical procedure or portion thereofand improves in performance. Where successful results for the use of amedical device are highly technique sensitive, the ability to “finetune” the user's response to prior instructions may help maintain theuser on the path to a successful outcome. For example, where a user“overcorrects” in response to a prior instruction, the MLM 600, inconjunction with the 3DGS 400, assists the user to further refine themovement to achieve a successful result.

To provide usable real time 3D AR feedback-based guidance to a medicaldevice user, the MLM 600 may include a standardized nomenclature module(not shown) to provide consistent real-time feedback instructions to theuser. In an alternative embodiment, multiple nomenclature options may beprovided to users, and different users may receive instructions thatvary based on the level of skill and background of the user. Forexample, users with an engineering background may elect to receive realtime feedback guidance from the machine learning module 600 and ARUI 300in in terminology more familiar to engineers, even where the user isperforming a medical task. Users with a scientific background may electto receive real time feedback guidance in terminology more familiar fortheir specific backgrounds. In some embodiments, or for some types ofequipment, however, a single, standardized nomenclature module may beprovided, and the machine learning module 600 may provide real timefeedback guidance using a single, consistent terminology.

The MLM 600 may also provide landmarks and virtual markings that areinformative to enable the user to complete the task, and the landmarksprovided in some embodiments may be standardized for all users, while inother embodiments different markers may be used depending upon thebackground of the user.

FIG. 3 illustrates a continuum of functionality of an ultrasound systemthat may include both standard ultrasound functionality in a first mode,in which no AR functions are used, as well as additional modes involvingAR functions. A second, “basic support” mode may also be provided with arelatively low level of Augmented Reality supplementation, e.g., anelectronic medical procedure display and fixed markers. A third mode,incorporating real-time, three-dimensional (3D) augmented reality (AR)feedback guidance, may also be selected.

In the embodiment of FIG. 2, MLM 600 provides outcome-based feedback bycomparing novice user ultrasound images and reference ultrasound imagesusing a neural network. The description provided herein of the use ofsuch neural networks is not intended to limit embodiments of the preventinvention to the use of neural networks, and other techniques may beused to provide outcome-based feedback.

A variety of neural networks may be used in MLM 600 to provideoutcome-based-feedback in a medical device system according to FIG. 1.Convolutional neural networks are often used in computer vision or imageanalysis applications. In systems involving image processing, such asFIG. 2, neural networks used in MLM 600 preferably include at least oneconvolutional layer, because image processing is the primary basis foroutcome-based feedback. In one embodiment, the neural network may beResNet, a neural network architecture developed by Microsoft Researchfor image classification. ResNet may be implemented in software using avariety of computer languages such as NDL, Python, or BrainScript. Inaddition to ResNet, other neural network architectures suitable forimage classification may also be used in different embodiments. Fordifferent medical equipment systems, or non-medical equipment, it willbe appreciated that other neural networks, having features moreapplicable to a different type of data generated by that equipment, maybe preferred.

In one embodiment of FIG. 2, ResNet may be used in the MLM 600 toclassify a continuous series of ultrasound images (e.g., at a desiredsampling rate such as 20 frames per second) generated by the novice user50 in real-time using ultrasound system 210. The images are classifiedinto groups based on whether the desired outcome is achieved, i.e.,whether the novice user's images match corresponding reference imageswithin a desired confidence level. The goal of classification is toenable the MLM to determine if the novice user's images capture theexpected view (i.e., similar to the reference images) of targetanatomical structures for a specified ultrasound medical procedure. Inone embodiment, the outcome-based feedback provided by the MLM 600includes 1) the most-probable identity of the ultrasound image (e.g.,the name of a desired structure such as “radial cross-section of thecarotid artery,” “lateral cross-section of the jugular vein,” etc.), and2) the probability of identification (e.g., 0% to 100%).

As an initial matter, ultrasound images from ultrasound system 210 mustbe converted to a standard format usable by the neural network (e.g.,ResNet). For example, ultrasound images captured by one type ofultrasound machine (FUS) are in the RGB24 image format, and may generateimages ranging from 512×512 pixels to 1024×768 pixels, depending on howthe ultrasound machine is configured for an ultrasound scan. During anyparticular scan, the size of all captured images will remain constant,but image sizes may vary for different types of scans. Neural networks,however, generally require that the images must be in a standardizedformat (e.g., CHW format used by ResNet) and a single, constant sizedetermined by the ML model. Thus, ultrasound images may need to beconverted into the standardized format. For example, images may beconverted for use in ResNet by extracting the CHW components from theoriginal RGB24 format to produce a bitmap in the CHW layout, as detailedathttps://docs.microsoft.com/en-us/cognitive-toolkit/archive/cntk-evaluate-image-transforms.It will be appreciated that different format conversion processes may beperformed by persons of skill in the art to produce images usable by aparticular neural network in a particular implementation.

Ultrasound medical procedures require the ultrasound user to capturespecific views of various desired anatomical structures from specificperspectives. These view/perspective combinations may be represented asclasses in a neural network. For example, in a carotid artery assessmentprocedure, the ultrasound user may be required to first capture theradial cross section of the carotid artery, and then capture the lateralcross section of the carotid artery. These two different views can berepresented as two classes in the neural network. To add additionaldepth, a third class can be used to represent any view that does notbelong to those two classes.

Classification is a common machine learning problem, and a variety ofapproaches have been developed. Applicants have discovered that a numberof specific steps are advisable to enable MLM 600 to have goodperformance in classifying ultrasound images to generate 3D AR feedbackguidance that is useful for guiding novice users. These include care inselecting both the training set and the validation data set for theneural network, and specific techniques for optimizing the neuralnetwork's learning parameters.

As noted, ResNet is an example of a neural network that may be used inMLM 600 to classify ultrasound images. Additional information on ResNetmay be found at https://arxiv.org/abs/1512.03385. Neural networks suchas ResNet are typically implemented in a program language such as NDL,Python, or BrainScript, and then trained using a deep machine learning(DML) platform or program such as CNTK, Caffe, or Tensorflow, amongother alternatives. The platform operates by performing a “trainingprocess” using a “training set” of image data, followed by a “validationprocess” using a “validation set” of image data. Image analysis ingeneral (e.g., whether part of the training and validation processes, orto analyze images of a novice user) is referred to as “evaluation” or“inferencing.”

In the training process, the DML platform generates a machine learning(ML) model using the training set of image data. The ML model generatedin the training process is then evaluated in the validation process byusing it to classify images from the validation set of image data thatwere not part of the training set. Regardless of which DML platform(e.g., CNTK, Caffe, Tensorflow, or other system) is used, the trainingand validation performance of ResNet should be is similar for a giventype of equipment (medical or non-medical). In particular, for theFlexible Ultrasound System (FUS) previously described, the imageanalysis performance of ResNet is largely independent of the DMLplatform.

In one embodiment, for small patient populations (e.g., astronauts,polar explorers, small maritime vessels), for each ultrasound procedure,a patient-specific machine learning model may be generated duringtraining using a training data set of images that are acquired during areference examination (e.g., by an expert) for each individual patient.Accordingly, during subsequent use by a novice user, for each particularultrasound procedure the images of a specific patient will be classifiedusing a patient-specific machine learning module for that specificpatient. In other embodiments, a single “master” machine learning modelis used to classify all patient ultrasound images. In patient-specificapproaches, less data is required to train the neural network toaccurately classify patient-specific ultrasound images, and it is easierto maintain and evolve such patient-specific machine learning models.

Regardless of which DML platform is used, the machine learning (ML)model developed by the platform has several common features. First, theML model specifies classes of images that input images (i.e., by anovice user) will be classified against. Second, the ML model specifiesthe input dimensions that determines the required size of input images.Third, the ML model specifies the weights and biases that determine theaccuracy of how input images will the classified.

The ML model developed by the DLM platform is the structure of theactual neural network that will be used in evaluating images captured bya novice user 50. The optimized weights and biases of the ML model areiteratively computed and adjusted during the training process. In thetraining process, the weights and biases of the neural network aredetermined through iterative processes known as Feed-Forward (FF) andBack-Propagation (BP) that involve the input of training data into aninput layer of the neural network and comparing the corresponding outputat the network's output layer with the input data labels until theaccuracy of the neural network in classifying images is at an acceptablethreshold accuracy level.

The quality of the training and validation data sets determines theaccuracy of the ML model, which in turn determines the accuracy of theneural network (e.g., ResNet) during image classification by a noviceuser. A high-quality data set is one that enables the neural network tobe trained within a reasonable time frame to accurately classify amassive variety of new images (i.e., those that do not appear in thetraining or validation data sets). Measures of accuracy and error forneural networks are usually expressed as classification error(additional details available athttps://www.gepsoft.com/gepsoft/APS3KB/Chapter09/Section2/SS01.htm),cross entropy error (https://en.wikipedia.org/wiki/Cross_entropy), andmean average precision(https://docs.microsoft.com/en-us/cognitive-toolkit/object-detection-using-fast-r-cnn-brainscript#map-mean-average-precision).

In one embodiment, the output of the neural network is the probability,for each image class, that an image belongs to the class. From thisoutput, the MLM 600 may provide output-based feedback to the novice userof one or both of 1) the best predicted class for the image (i.e., theimage class that the neural network determines has the highestprobability that the image belongs to the class), and 2) the numericalprobability (e.g., 0% to 100%) of the input image belonging to the bestpredicted class. The best predicted class may be provided to the noviceuser in a variety of ways, e.g., as a virtual text label, while thenumerical probability may also be displayed in various ways, e.g., as anumber, a number on a color bar scale, as a grayscale color varyingbetween white and black, etc.

To train a neural network such as ResNet to classify ultrasound imagesfor specific ultrasound procedures performed with ultrasound system 210,many high quality images are required. In many prior art neural networkapproaches to image classification, these data sets are manuallydeveloped in a highly labor-intensive process. In one aspect, thepresent disclosure provides systems and methods for automating one ormore portions of the generation of training and validation data sets.

Using software to automate the process of preparing accurately labeledimage data sets not only produces data sets having minimal or noduplicate images, but also enables the neural network to be continuouslytrained to accurately classify large varieties of new images. Inparticular, automation using software allows the continual generation orevolution of existing image data sets, thereby allowing the continualtraining of ResNet as the size of the image data set grows over time. Ingeneral, the more high-quality data there is to train a neural network,the higher the accuracy of the neural network's ability to classifyimages will be. This approach contrasts sharply with the manualapproaches to building and preparing image data sets for deep machinelearning.

As one nonlimiting example, an ultrasound carotid artery assessmentprocedure requires at least 10,000 images per patient for training apatient-specific neural network used to provide outcome-based feedbackto a novice user in a 3D AR medical guidance system of the presentdisclosure. Different numbers of images may be used for differentimaging procedures, with the number of images will depending upon theneeds of the particular procedure.

The overall data set is usually split into two subsets, with 70-90%,more preferably 80-85%, of the images being included as part of atraining set and 10-30%, more preferably 15-20%, of the images includedin the validation data set, with each image being used in only one ofthe two subsets (i.e., for any image in the training set, no duplicateof it should exist in the validation set. In addition, any excessivenumber of redundant images in the training set should be removed toprevent the neural network from being overfitted to a majority ofidentical images. Removal of such redundant images will improve theability of the neural network to accurately classify images in thevalidation set. In one embodiment, an image evaluation module evaluateseach image in the training set to determine if it is a duplicate ornear-duplicate of any other image in the database. The image evaluationmodule computes each image's structural similarity index (SSI) againstall other images in the set. If the SSI between two images is greaterthan a similarity threshold, which in one nonlimiting example may beabout 60%, then the two images are regarded as near duplicates and theimage evaluation module removes all one of the duplicate or nearduplicate images. Further, images that are down to exist both in thetraining set and the validation set are likewise removed (i.e., theimage evaluation module computes SSI values for each image in thetraining set against each image in the validation set, and removesduplicate or near-duplicate images from one of the training andvalidation sets). The reduction of duplicate images allows the neuralnetwork to more accurately classify images in the validation set, sincethe chance of overfitting the neural network during training to amajority of identical images is reduced or eliminated.

FIG. 6 illustrates a method 602 for developing a ML model for training aneural network using manually prepared data sets. First, a referenceuser (e.g., an expert sonographer or ultrasound technician) captures(610) all the necessary ultrasound views of the target anatomicalstructures for the ultrasound carotid artery assessment (or medicalprocedure), including 10,000 or more images. The population size of eachview or class should be equal. For the carotid artery assessment, theradial, lateral, and unknown views are captured, which is around 3,300+images per view or class.

Next the reference user manually labels (615) each image as one of theavailable classes. For the carotid artery assessment, the images arelabeled as radial, lateral or unknown.no image overlap in the trainingand validation data sets). For each labeled image, the reference usermay in some embodiments (optional), manually identify (620) the exactarea within the image where the target anatomical structure is located,typically with a box bounding the image. Two examples of this the use ofbounding boxes to isolate particular structures are provided in FIGS.10A and 10B, which shows the location of a carotid artery within anultrasound image.

Once the entire data set is properly labeled, it is manually split (625)into the training data set and the validation data sets, which may thenbe used to train the neural network (e.g., ResNet). Neural networkscomprise a series of coupled nodes organized into at least an input andan output layer. Many neural networks have one or more additional layers(commonly referred to as “hidden layers”) that may include one or moreconvolutional layers as previously discussed regarding MLM 600.

The method 600 also comprises loading (630) the neural networkdefinition (such as a definition of ResNet), usually expressed as aprogram in a domain-specific computer language such as NDL, Python orBrainScript, into a DML platform or program such as CNTK, Caffe orTensorflow. The DML platforms offer tunable or adjustable parametersthat are used to control the outcome of the training process. Some ofthe parameters are common to all DML platforms, such as types of loss orerror, accuracy metrics, types of optimization or back-propagation(e.g., Stochastic Gradient Descent and Particle Swarm Optimization).Some adjustable parameters are specific to one or more of the foregoing,such as parameters specific to Stochastic Gradient Descent such as thenumber of epochs to train, training size (e.g., minibatch), learningrate constraints, and others known to persons of skill in the art. Inone example involving CNTK as the DML platform, the adjustableparameters include learning rate constraints, number of epochs to train,epoch size, minibatch size, and momentum constraints.

The neural network definition (i.e., a BrainScript program of ResNet)itself also has parameters that may be adjusted independently of anyparameter adjustments or optimization of parameters in the DML platform.These parameters are defined in the neural network definition such asthe connections between deep layers, the types of layers (e.g.,convolutional, max pooling, ReLU), and their structure/organization(e.g., dimensions and strides). If there is minimal error or highaccuracy during training and/or validating, then adjustment of theseparameters may have a lesser effect on the overall image analysisperformance compared to adjusting parameters not specific to the neuralnetwork definition (e.g., DML platform parameters), or simply having ahigh quality training data set. In the case of a system developed forcarotid artery assessment, no adjustments to the neural networkparameters were needed to achieve less than 10%-15% error, in thepresence of a high quality training data set.

Referring again to FIG. 6, the methods also includes (635) feeding thetraining data set into the DML platform and performing the trainingprocess (640). After the training process is completed, training processmetrics for loss, accuracy and/or error are obtained (645). Adetermination is made (650) whether the training process metrics arewithin an acceptable threshold for each metric. If the training processmetrics are outside of an acceptable threshold for the relevant metrics,the adjustable parameters are adjusted to different values (655) and thetraining process is restarted (640). Parameter adjustments may be madeone or more times. However, if the training process 640 fails to yieldacceptable metrics (650) after a threshold number of iterations orrepetitions (e.g., two, three or another number), then the data set isinsufficient to properly train the neural network and it is necessary toregenerate the data set. If the metrics are within an acceptablethreshold for each metric, then a ML model has been successfullygenerated (660). In one embodiment, acceptable thresholds may range fromless than 5% to less than 10% average cross-entropy error for allepochs, and from less than 15% to less than 10% average classificationerror for all epochs. If will be recognized that different developmentprojects may involve different acceptable thresholds.

The method then includes feeding the validation data set to the ML model(665), and the validation process is performed (670) using thevalidation data set. After the completion of the validation process,validation process metrics for loss, accuracy and/or error are obtained(675) for the validation process. A determination is made (680) whetherthe validation metrics are within an acceptable threshold for eachmetric, which may be the same as or different from those used for thetraining process. If the validation process metrics are outside of theacceptable thresholds, the adjustable parameters are adjusted todifferent values (655) and the training process is restarted (640). Ifthe metrics are acceptable, then the ML model may be used to classifynew data (685).

The process may be allowed to continue through one or more additionalcycles. If validation process metrics are still unacceptable, then thedata set is insufficient to properly train the neural network, and thedata set needs to be regenerated.

Referring again to FIG. 6, the initial portions of the process arehighly labor-intensive. Specifically, the steps of capturing ultrasoundimages (610), manually labeling (615) and identifying target areas areusually performed at great cost in time and expense by a reference user(e.g., a sonographer or ultrasound technician, nurse, or physician). Inaddition, splitting the data set into training and validation sets mayalso involve significant manual discretion by the reference user.

In one aspect, the present invention involves using computer software toautomate or significantly speed up one or more of the foregoing steps.Although capturing ultrasound images during use of the ultrasound systemby a reference or expert user (610) necessarily requires the involvementof an expert, in one embodiment the present disclosure includes systemsand methods for automating all or portions of steps 610-625 of FIG. 6.

FIG. 7 illustrates a machine learning development module (MLDM) 705 forautomating some or all of the steps of developing training andvalidation image data sets for a particular medical imaging procedure,in this instance a carotid artery assessment procedure. I will beunderstood that multiple MLDMs, different from that shown in FIG. 7, maybe provided for each imaging procedure for which 3D AR feedback is to beprovided by a system according to FIG. 1. Manually capturing, labeling,isolating, and dividing the images into a two image data sets is notonly time consuming and expensive, but is also error prone because ofthe subjective judgment that must be exercised by the reference user inlabeling and isolating the relevant portions of each image captured fora given procedure. The accuracy and speed of these processes may beimproved using automated image processing techniques to provideconsistent analysis of the image patterns of target anatomicalstructures specific to a particular ultrasound medical procedure.

In one embodiment, MLDM 705 is incorporated into computer system 700(FIG. 1) and communicates with an imaging medical equipment system(e.g., an ultrasound system 210, FIG. 2). Referring again to FIG. 7,MLDM 705 includes an image capture module 710 that may automaticallycapture images from the ultrasound system 210 while a reference userperforms a carotid artery assessment associated with MLDM 705 (or adifferent procedure associated with a different MLDM). The image capturemodule 710 comprises one or more of hardware, firmware, software or acombination thereof, in computer 700 (FIG. 1).

Image capture module 710 may also comprise an interface such as agraphical user interface (GUI) 712 for display on a screen of computer700 or ultrasound system 210. The GUI 712 may permit an operator (e.g.,the reference user or a system developer) to automatically captureimages while the reference user performs the medical procedure specificto MLDM 705 (e.g., a carotid artery assessment). More specifically, theGUI 712 enables a user to program the image capture module 710 tocapture images automatically (e.g., at a specified time interval such as10 Hz, or when 3DGS 400 detects that probe 210 is at a particularanatomical position) or on command (e.g., by a capture signal activatedby the operator using a sequence of keystrokes on computer 700 or abutton on ultrasound probe 215). The GUI 712 allows the user to definethe condition(s) under which images are captured by image capture module710 while the reference user performs the procedure of MLDM 705.

Once images have been captured (e.g., automatically or on command) byimage capture module 710, MLDM 705 includes one or more feature modules(715, 720, 725, 745, etc.) to identify features associated with thevarious classes of images that are available for the procedure of MLDM705. The features may be aspects of particular structures that definewhich class a given image should belong to. Each feature module definesthe image criteria to determine whether a feature is present in theimage. Depending on the number of features and the number of classes(which may each contain multiple features, MLDMs for different imagingprocedures may have widely different numbers of feature modules.Referring again to FIG. 7, MLDM 705 applies each of the feature modulesfor the procedure to each image captured for that procedure to determineif and where the features are present in each captured image. An exampleof various features and how they may be defined in the feature modulesis provided in FIGS. 9A-9G, discussed more fully below.

For example, in a carotid artery assessment procedure, the availableclasses may include a class of “radial cross section of the carotidartery,” a class of “lateral cross section of the carotid artery,” and aclass of “unknown” (or “neither radial cross section nor lateral crosssection”). For an image to be classified as belonging to the “radialcross section of the carotid artery” class, various features associatedwith the presence of the radial cross section of a carotid artery mustbe present in the image. The feature modules, e.g., 715, 720, etc., areused by the MLDM 705 to analyze captured images to determine whether agiven image should be placed in the class of “radial cross section ofthe carotid artery” or in another class. Because the feature modules areeach objectively defined, the analysis is less likely to be mislabeledbecause of the reference user's subjective bias.

Finally, each MLDM 705 may include a classification module 750 toclassify each of the captured images with a class among those availablefor MLDM 705. Classification module 750 determines the class for eachimage based on which features are present and not present in the image,and labels each image as belonging to the determined class. Because thefeature modules are each objectively defined, the classification module750 is less likely to mislabel images than manual labeling based on thesubjective judgment exercised by the reference user.

Computer 700 (FIG. 1) may include a plurality of MLDMs similar to module705, each of which enables automating the process of capturing andlabeling images for a different imaging procedure. It will beappreciated that different modules may be provided for automating thecapture and labeling of data from different types of medical ornon-medical equipment during their use by a reference user or expert. Inone alternative embodiment, a central library (e.g., library 500,FIG. 1) of features may be maintained for all procedures for which 3D ARguidance to a novice user are to be provided by a system 100 of FIG. 1.In such an embodiment, the features (whether software, firmware, orhardware) are maintained separately from computer 700, and the structureof MLDMs such as MLDM 705 may be simplified such that each MLDM simplyaccesses or calls the feature modules for its particular procedure fromthe central feature library.

The automated capture and labeling of reference data by MLDM 705 may bebetter understood by an example of a carotid artery assessment using anultrasound system. The radial and lateral cross-sections of the carotidartery have distinct visual features that can be used to identify theirpresence it ultrasound images at specific ultrasound depths. Thesevisual features or criteria may be defined and stored as feature modules715, 720, 725, etc. in MLDM 705 (or a central feature library inalternative embodiments) for a carotid artery assessment procedure.Captured images are then analyzed using the feature modules determinewhether or not each of the carotid artery assessment features arepresent. The presence or absence of the features are then used toclassify each image into one of the available classes for the carotidartery assessment procedure.

The feature modules 715, 720, 725, etc. provide consistent analysis ofimage patterns of the target anatomical structures in the imagescaptured during a reference carotid artery assessment procedure (e.g.,by an expert). Feature modules for each image class may be defined by areference user, a system developer, or jointly by both, for any numberof ultrasound procedures such as the carotid artery assessmentprocedure.

Once the features for each carotid artery assessment procedure imageclass have been defined and stored as feature modules 715, 720, 725,etc., standard image processing algorithms (e.g., color analysisalgorithms, thresholding algorithms, convolution with kernels, contourdetection and segmentation, clustering, and distance measurements) areused in conjunction with the defined features to identify and measurewhether the features are present in the captured reference images. Inthis way, the feature modules allow the MLDM 705 to automate (fully orpartially) the labeling of large data sets in a consistent andquantifiable manner.

The visual feature image processing algorithms, in one embodiment, areperformed on all of the images that are captured during the referenceperformance of the particular medical procedure associated with thefeature module, using software, firmware and/or hardware. The ability ofthe labeling module to label images may be verified by review of theautomated labeling of candidate images by a reference user (e.g., anexpert sonographer, technician, or physician). The foregoing processesand modules allow developers and technicians to quickly and accuratelylabel and isolate target structures in large image data sets of 10,000or more images.

MLDMs as shown in FIG. 7 facilitate consistent labeling because thevisual features are determined numerically by standard algorithms afterbeing defined by a reference user, expert, or system developer. Theautomated labeling is also quantified, because the features aredetermined numerically according to precise definitions.

Although the functions and operation of MLDM 705 have been illustratedfor a carotid artery assessment ultrasound procedure, it will beappreciated that additional modules (not shown) may be provided fordifferent ultrasound procedures (e.g., a cardiac assessment procedure ofthe heart), and that such modules would include additional class andfeatures modules therein. In addition, for non-imaging types of medicalequipment, e.g., an EKG machine, labeling modules may also be providedto classify the output of the EKG machine into one or more classes(e.g., heart rate anomalies, QT interval anomalies, R-wave anomalies,etc.) having different structures and analytical processes but a similarpurpose of classifying the equipment output into one or more classes.

Applicants have discovered that the automated capture and labeling ofreference image data sets may be improved by automatically adjustingcertain parameters within the feature modules 715, 720, 725, etc. Aspreviously noted, the features modules use standard image processingalgorithms to determine whether the defined features are present in eachimage. These image processing algorithms in the feature modules (e.g.,color analysis algorithms, thresholding algorithms, convolution withkernels, contour detection and segmentation, clustering and distancemeasurements) include a number of parameters that are usually maintainedas constants, but which may be adjusted. Applicants have discovered thatby automatically optimizing these adjustable parameters within the imageprocessing algorithms using Particle Swarm Optimization, it is possibleto minimize the number of mislabeled images by the image processingalgorithms in the features modules. Automatic adjustment of the featuremodules analysis image processing algorithms is discussed more fully inconnection with FIG. 8.

FIG. 8 illustrates one embodiment of a method 802 for developing amachine learning (ML) model of a neural network for classifying imagesfor a medical procedure using automatically prepared data sets for anultrasound system. In one embodiment, the method may be performed usinga system according to FIG. 1 that incorporates the machine learningdevelopment module (MLDM) 705 of FIG. 7. In alternative embodiments, themethod may be implemented for different types of medical or non-medicalequipment.

The method includes automatically capturing a plurality of ultrasoundimages (805) during a reference ultrasound procedure (e.g., performed byan expert), wherein each of the plurality of images is capturedaccording to defined image capture criteria. In one embodiment, capturemay be performed by an image capture module implemented in a computer(e.g., computer 700, FIG. 1) in one or more of software, firmware, orhardware, such as image capture module 710 and GUI 712 (FIG. 7).

Referring again to FIG. 8, the method further comprises automaticallyanalyzing each image to determine whether one or more features ispresent in each image (810). The features correspond to those present inone or more image classes, and the presence or absence of certainfeatures may be used to classify a given image in one or more imageclasses for the reference medical procedure. A plurality of featuremodules (e.g., feature modules 715, 720, etc. of FIG. 7) stored in amemory may be used to analyze the images for the presence or absence ofeach feature. The feature modules may comprise software, firmware, orhardware, and a computer such as computer 700 of FIG. 1 may analyzeimage captured image using the feature modules.

The method further comprises automatically classifying and labeling(815) each image as belonging to one of a plurality of available classesfor the ultrasound medical procedure. As noted above, each image may beassigned to a class based on the features present or absent from theimage. After an image is classified, the method further compriseslabeling the image with its class. Labeling may be performed by storingin memory the image's class, or otherwise associating the result of theclassification process with the image in a computer memory. In oneembodiment, image classification may be performed by a classificationmodule such as classification module 750 of FIG. 7. Labeling may beperformed by the classification module that classifies the image, or bya separate labeling module.

In some embodiments, the method may also involve automatically isolating(e.g., using boxes, circles, highlighting or other designation) withineach image where each feature (i.e., those determined to be present inthe feature analysis step) is located within the image (820). This stepis optional and may not be performed in some embodiments. In oneembodiment, automatic feature isolation (or bounding) may be performedby an isolation module that determines the boundary of each featurebased on the characteristics that define the feature. The isolationmodule may apply appropriate boundary indicators (e.g., boxes, circles,ellipses, etc.) as defined in the isolation module, which in someembodiments may allow a user to select the type of boundary indicator tobe applied.

After the images have been classified and labeled, the method includesautomatically splitting the set of labeled images into a training setand a validation set (825). The training set preferably is larger thanthe validation set (i.e., comprises more than 50% of the total images inthe data set), and may range from 70-90%, more preferably 80-85%, of thetotal images. Conversely, the validation set may comprise from 10-30,more preferably from 15-20%, of the total images.

The remaining steps in the method 802 (e.g., steps 830-885) areautomated steps that are similar to corresponding steps 630-685 andwhich, for brevity, are described in abbreviated form. The methodfurther comprises providing a Deep Machine Learning (DML) platform(e.g., CNTK, Caffe, or Tensorflow) having a neural network to be trainedloaded onto it (830). More specifically, a neural network (e.g., ResNet)is provided as a program in a computer language such as NDL or Python inthe DML platform.

The training set is fed into the DML platform (835) and the trainingprocess is performed (840). The training process comprises iterativelycomputing weights and biases for the nodes of the neural network usingfeed-forward and back-propagation, as previously described, until theaccuracy of the network in classifying images reaches an acceptablethreshold level of accuracy.

The training process metrics of loss, accuracy, and/or error areobtained (845) at the conclusion of the training process, and adetermination is made (850) whether the training process metrics arewithin an acceptable threshold for each metric. If the training processmetrics are unacceptable, the adjustable parameters of the DML platform(and optionally those of the neural network) are adjusted to differentvalues (855) and the training process is restarted (840). In one exampleinvolving CNTK as the DML platform, the tunable or adjustable parametersinclude learning rate constraints, number of epochs to train, epochsize, minibatch size, and momentum constraints.

The training process may be repeated one or more times if error metricsare not acceptable, with new adjustable parameters being provided eachtime the training process is performed. In one embodiment, if the errormetrics obtained for the training process are unacceptable, adjustmentsto the adjustable parameters (855) of the DML platform are madeautomatically, using an optimization technique such as Particle SwarmOptimization. Additional details on particle swarm theory are providedby Eberhart, R. C. & Kennedy, J., “A New Optimizer Using Particle SwarmTheory,” Proceedings of the Sixth International Symposium on MicroMachine and Human Science, 39-43 (1995). In another embodiment,adjustments to the adjustable parameters (855) in the event ofunacceptable error metrics are made manually by a designer.

In one embodiment, each time automatic adjustments are made (855) to theadjustable parameters of the DML platform, automatic adjustments arealso made to the adjustable parameters of the image processingalgorithms used in the feature modules. As discussed in connection withFIG. 7, standard image processing algorithms (e.g., color analysisalgorithms, thresholding algorithms, convolution with kernels, contourdetection and segmentation, clustering and distance measurements)include a number of parameters that are usually maintained as constants,but which may be adjusted. In a particular embodiment, the step ofadjusting the adjustable parameters of the DML platform comprisesautomatically adjusting at least one of the adjustable parameters of theDML platform and automatically adjusting at least one of the adjustableparameters of the image processing algorithms. In a still more specificembodiment, Particle Swarm Optimization is used to automatically adjustboth at least one adjustable parameter of the DML platform and at leastone adjustable parameter of an image processing algorithm.

If the training process 840 fails to yield acceptable metrics (650)after a specific number of iterations (which may be manually determined,or automatically determined by, e.g., Particle Swarm Optimization), thenthe data set is insufficient to properly train the neural network andthe data set is regenerated. If the metrics are within an acceptablethreshold for each metric, then a DML model has been successfullygenerated (860). In one embodiment, acceptable error metrics may rangefrom less than 5% to less than 10% average cross-entropy error for allepochs, and from less than 50% to less than 10% average classificationerror for all epochs. If will be recognized that different developmentprojects may involve different acceptable thresholds, and that differentDML platforms may use different types of error metrics.

If a successful DML model is generated (860), the method then includesfeeding the validation data set to the DML model (865), and thevalidation process is performed (870) using the validation data set.After the completion of the validation process, validation processmetrics for loss, accuracy and/or error are obtained (875) for thevalidation process.

A determination is made (880) whether the validation metrics are withinan acceptable threshold for each metric, which may be the same as ordifferent from those used for the training process. If the validationprocess metrics are outside of the acceptable threshold, the adjustableparameters are adjusted to different values (855) and the trainingprocess is restarted (840). If the metrics are acceptable, then the DMLmodel may be used to classify new data (885). In one embodiment, thestep of adjusting the adjustable parameters of the DML platform afterthe validation process comprises automatically adjusting at least one ofthe adjustable parameters of the DML platform and automaticallyadjusting at least one of the adjustable parameters of the imageprocessing algorithms, for example by an algorithm using Particle SwarmOptimization.

The process may be allowed to continue through one or more additionalcycles. If evaluation process metrics are still unacceptable, then thedata set is insufficient to properly train the neural network, and thedata set needs to be regenerated.

FIGS. 9A-9G are examples of features that may be used to classify imagesinto the class of “radial cross section of the carotid artery.” In someembodiments, ultrasound systems capable of providing color data may beused, and systems of the present invention may provide outcome-basedfeedback from color data in captured images. Although rendered ingrayscale for simplicity, FIGS. 9A and 9B illustrates an image of acarotid artery processed to identify colors using the HSV color space,although in alternative embodiments color may be represented as valuesin other color space schemes such as RGB. Persons of skill in the art ofprocessing color ultrasound images will appreciate that bright colorintensity in several areas suggests the presence of blood flow,especially in the lighter blue and lighter turquoise areas (FIG. 9A) andthe white areas (FIG. 9B) of the V channel of the HSV color space. Inalternative embodiments, ultrasound systems capable of only grayscaleimages may be used.

FIG. 9C was obtained by processing the image of FIG. 9A using adaptedthresholding and Canny edge detection to identify the general contour ofthe arterial wall, with the contours being represented as edges in agraphical figure. FIG. 9C illustrates a generally circular area in thecenter-right area of the figure that suggests the possibility of aradial cross-section of the carotid artery. A linear area on the lowerleft suggests the possibility of bright artifacts that are of littleinterest.

FIG. 9D was obtained by processing the image of FIG. 9A using clusteringto identify clusters of contours, and isolate the single cluster ofcontours that match the general area of the lumen of the artery. Thegenerally elliptical area in the center-right is the single cluster ofcontours that match the general area and geometry of the radial crosssection of the carotid artery, while the three clusters are merelyartifacts or noise that do not match the general area or geometry of theaforementioned cross section.

FIGS. 9E is a generalization of FIG. 9D using the centers of mass foreach cluster to show how clusters are expected to be positioned relativeto each other. The clusters are represented as sets of points in 2Dspace. Proximity is represented as vectors.

FIG. 9F uses known anatomical markers, such as cross sections of veinsor bones, and expected relative positions to verify structures. Inparticular, the right-side portion of FIG. 9F shows the bright radialcross section of the carotid artery as processed in FIG. 9B, and iscompared to the left-side portion of FIG. 9F, which shows the same imageprocessed using binary thresholding to better illustrate (upper darkelliptical region in large white area) where the nearby jugular veinwould be. This illustrates the expected proximity of the artery relativeto the vein, and confirms the position of the artery shown in FIG. 9E.

As discussed in connection with FIGS. 6 and 8, preparation of the imagesfor the neural network training and validation data sets in someembodiments includes isolating or visually indicating in the imageswhere features are located. Isolating involves applying boundaryindicators, such as a bounding box, circle, ellipse, or other regular orirregular bounding shape or region, around the feature of interest. Inone embodiment (FIG. 6, step 820) this optional step may be performedmanually by an expert as part of the manual process of preparing thedata sets for training the neural network. In another embodiment (FIG.8, step 820), automatic feature isolation (or bounding) may be performedautomatically by an isolation module that determines the boundary ofeach feature based on the characteristics that define the feature.

Examples of isolating boxes are shown in FIGS. 10A and 10B. FIG. 10Ashows a manually generated bounding box to indicate the presence of alateral view of a carotid artery. FIG. 10B illustrates a manuallygenerated bounding box to indicate the presence of a cross-sectionalview of a carotid artery.

In various embodiments, the present invention relates to the subjectmatter of the following numbered paragraphs.

101. A method for providing real-time, three-dimensional (3D) augmentedreality (AR) feedback guidance to a user of a medical equipment system,the method comprising:

-   -   receiving data from a medical equipment system during a medical        procedure performed by a user of the medical equipment to        achieve a medical procedure outcome;    -   sensing real-time user positioning data relating to one or more        of the movement, position, and orientation of at least a portion        of the medical equipment system within a volume of the user's        environment during the medical procedure performed by the user;    -   retrieving from a library at least one of 1) stored reference        positioning data relating to one or more of the movement,        position, and orientation of at least a portion of the medical        equipment system during reference a medical procedure, and 2)        stored reference outcome data relating to a reference        performance of the medical procedure;    -   comparing at least one of 1) the sensed real-time user        positioning data to the retrieved reference positioning data,        and 2) the data received from the medical equipment system        during a medical procedure performed by the user to the        retrieved reference outcome data;    -   generating at least one of 1) real-time position-based 3D AR        feedback based on the comparison of the sensed real-time user        positioning data to the retrieved reference positioning data,        and 2) real-time output-based 3D AR feedback based on the        comparison of the data received from the medical equipment        system during a medical procedure performed by the user to the        retrieved reference outcome data; and    -   providing at least one of the real-time position-based 3D AR        feedback and the real-time output-based 3D AR feedback to the        user via an augmented reality user interface (ARUI).

102. The method of claim 101, wherein the medical procedure performed bya user of the medical equipment comprises a first medical procedure, andthe stored reference positioning data and stored reference outcome datarelate to a reference performance of the first medical procedure priorto the user's performance of the first medical procedure.

103. The method of claim 101, wherein the medical procedure performed bya user of the medical equipment comprises a first ultrasound procedure,and the stored reference positioning data and stored reference outcomedata comprise ultrasound images obtained during a reference performanceof the first ultrasound procedure prior to the user's performance of thefirst ultrasound procedure.

104. The method of claim 103, wherein sensing real-time user positioningdata comprises sensing real-time movement by the user of an ultrasoundprobe relative to the body of a patient.

105. The method of claim 101, wherein generating real-time outcome-based3D AR feedback is based on a comparison, using a neural network, ofreal-time images generated by the user in an ultrasound procedure toretrieved images generated during a reference performance of the sameultrasound procedure prior to the user.

106. The method of claim 105, wherein the comparison is performed by aconvolutional neural network.

107. The method of claim 101, wherein sensing real-time user positioningdata comprises sensing one or more of the movement, position, andorientation of at least a portion of the medical equipment system by theuser with a sensor comprising at least one of a magnetic GPS system, adigital camera tracking system, an infrared camera system, anaccelerometer, and a gyroscope.

108. The method of claim 101, wherein sensing real-time user positioningdata comprises sensing at least one of:

-   -   a magnetic field generated by said at least a portion of the        medical equipment system;    -   the movement of one or more passive visual markers coupled to        one or more of the patient, a hand of the user, or a portion of        the medical equipment system; and    -   the movement of one or more active visual markers coupled to one        or more of the patient, a hand of the user, or a portion of the        medical equipment system.

109. The method of claim 101, wherein providing at least one of thereal-time position-based 3D AR feedback and the real-time output-based3D AR feedback to the user comprises providing a feedback selected from:

-   -   a virtual prompt indicating a movement correction to be        performed by a user;    -   a virtual image or video instructing the user to change the        orientation of a probe to match a desired orientation;    -   a virtual image or video of a correct motion path to be taken by        the user in performing a medical procedure;    -   a color-coded image or video indicating correct and incorrect        portions of the user's motion in performing a medical procedure;    -   and instruction to a user to press an ultrasound probe deeper or        shallower into tissue to focus the ultrasound image on a desired        target structure of the patient's body;    -   an auditory instruction, virtual image, or virtual video        indicating a direction for the user to move an ultrasound probe;        and    -   tactile information.

110. The method of claim 101, wherein providing at least one of thereal-time position-based 3D AR feedback and the real-time output-based3D AR feedback comprises providing both of the real-time position-based3D AR feedback and the real-time output-based 3D AR feedback to theuser.

111. The method of claim 101, wherein providing at least one of thereal-time position-based 3D AR feedback and the real-time output-based3D AR feedback comprises providing said at least one feedback to a headmounted display (HMD) worn by the user.

201. A method for developing a machine learning model of a neuralnetwork for classifying images for a medical procedure using anultrasound system, the method comprising:

-   -   A) performing a first medical procedure using an ultrasound        system;    -   B) automatically capturing a plurality of ultrasound images        during the performance of the first medical procedure, wherein        each of the plurality of ultrasound images is captured at a        defined sampling rate according to defined image capture        criteria;    -   C) providing a plurality of feature modules, wherein each        feature module defines a feature which may be present in an        image captured during the medical procedure;    -   D) automatically analyzing each image using the plurality of        feature modules;    -   E) automatically determining, for each image, whether or not        each of the plurality of features is present in the image, based        on the analysis of each imagine using the feature modules;    -   F) automatically labeling each image as belonging to one class        of a plurality of image classes associated with the medical        procedure;    -   G) automatically splitting the plurality of images into a        training set of images and a validation set of images;    -   H) providing a deep machine learning (DML) platform having a        neural network to be trained loaded thereon, the DML platform        having a plurality of adjustable parameters for controlling the        outcome of a training process;    -   I) feeding the training set of images into the DML platform;    -   J) performing the training process for the neural network to        generate a machine learning model of the neural network;    -   K) obtaining training process metrics of the ability of the        generated machine learning model to classify images during the        training process, wherein the training process metrics comprise        at least one of a loss metric, an accuracy metric, and an error        metric for the training process;    -   L) determining whether each of the at least one training process        metrics is within an acceptable threshold for each training        process metric;    -   M) if one or more of the training process metrics are not within        an acceptable threshold, adjusting one or more of the plurality        of adjustable DML parameters and repeating steps J, K, and L;    -   N) if each of the training process metrics is within an        acceptable threshold for each metric, performing a validation        process using the validation set of images;    -   O) obtaining validation process metrics of the ability of the        generated machine learning model to classify images during the        validation process, wherein the validation process metrics        comprise at least one of a loss metric, an accuracy metric, and        an error metric for the validation process;    -   P) determining whether each of the validation process metrics is        within an acceptable threshold for each validation process        metric;    -   Q) if one or more of the validation process metrics are not        within an acceptable threshold, adjusting one or more of the        plurality of adjustable DML parameters and repeating steps J-P;        and    -   R) if each of the validation process metrics is within an        acceptable threshold for each metric, storing the machine        learning model for the neural network.

202. The method of claim 201, further comprising:

-   -   S) receiving, after storing the machine learning model for the        neural network, a plurality of images from a user performing the        first medical procedure using an ultrasound system;    -   T) using the stored machine learning model to classify each of        the plurality of images received from the ultrasound system        during the second medical procedure.

203. The method of claim 201, further comprising:

-   -   S) using the stored machine learning model for the neural        network to classify a plurality of ultrasound images for a user        performing the first medical procedure.

204. The method of claim 201, wherein performing the training processcomprises iteratively computing weights and biases for each of the nodesof the neural network using feed-forward and back-propagation until theaccuracy of the network in classifying images reaches an acceptablethreshold level of accuracy.

205. The method of claim 201, wherein performing the validation processcomprises using the machine learning model generated by the trainingprocess to classify the images of the validation set of image data.

206. The method of claim 201, further comprising stopping the method ifsteps J, K, and L have been repeated more than a threshold number ofrepetitions.

207. The method of claim 206, further comprises stopping the method ifsteps N-Q have been repeated more than a threshold number ofrepetitions.

208. The method of claim 201, wherein providing a deep machine learning(DML) platform comprises providing a DML platform having at least oneadjustable parameter selected from learning rate constraints, number ofepochs to train, epoch size, minibatch size, and momentum constraints.

209. The method of claim 208, wherein adjusting one or more of theplurality of adjustable DML parameters comprises automatically adjustingsaid one or more parameters using a particle swarm optimizationalgorithm.

210. The method of claim 201, wherein automatically splitting theplurality of images comprises automatically splitting the plurality ofimages into a training set comprising from 70% to 90% of the pluralityof images, and a validation set comprising from 10% to 30% of theplurality of images.

211. The method of claim 201, wherein automatically labeling each imagefurther comprises isolating one or more of the features present in theimage using a boundary indicator selected from a bounding box, abounding circle, a bounding ellipse, and an irregular bounding region.

212. The method of claim 201, wherein obtaining training process metricscomprises obtaining at least one of average cross-entropy error for allepochs and average classification error for all epochs.

213. The method of claim 201, wherein determining whether each of thetraining process metrics are within an acceptable threshold comprisesdetermining whether average cross-entropy error for all epochs is lessthan a threshold selected from 5% to 10%, and average classificationerror for all epochs is less than a threshold selected from 15% to 10%.

214. The method of claim 201, wherein step A) is performed by an expert.

The particular embodiments disclosed above are illustrative only, as theinvention may be modified and practiced in different but equivalentmanners apparent to those skilled in the art having the benefit of theteachings herein. Examples are all intended to be non-limiting.Furthermore, exemplary details of construction or design herein shownare not intended to limit or preclude other designs achieving the samefunction. It is therefore evident that the particular embodimentsdisclosed above may be altered or modified and all such variations areconsidered within the scope and spirit of the invention, which arelimited only by the scope of the claims.

Embodiments of the present invention disclosed and claimed herein may bemade and executed without undue experimentation with the benefit of thepresent disclosure. While the invention has been described in terms ofparticular embodiments, it will be apparent to those of skill in the artthat variations may be applied to systems and apparatus described hereinwithout departing from the concept, spirit and scope of the invention.

What is claimed is:
 1. A medical guidance system (100) for providingreal-time, three-dimensional (3D) augmented reality (AR) feedbackguidance in the use of a medical equipment system (200), the medicalguidance system comprising: a medical equipment interface to a medicalequipment system (200), wherein said medical equipment interface iscapable of receiving data from the medical equipment system during amedical procedure performed by a user; an augmented reality userinterface (ARUI) (300) for presenting data pertaining to both real andvirtual objects to the user during at least a portion of the performanceof the medical procedure; a three-dimensional guidance system (3DGS)(400) that is capable of sensing real-time user positioning datarelating to one or more of the movement, position, and orientation of atleast a portion of the medical equipment system (200) during saidmedical procedure performed by the user; a library (500) containing 1)stored reference positioning data relating to one or more of themovement, position, and orientation of at least a portion of the medicalequipment system (200) during a reference medical procedure and 2)stored reference outcome data relating to an outcome of said referencemedical procedure; and a machine learning module (MLM) (600) forproviding at least one of 1) position-based 3D AR feedback to the userbased on the sensed user positioning data and the reference positioningdata, and 2) outcome-based 3D AR feedback to the user based on datareceived from the medical equipment system during the medical procedureperformed by the user and reference outcome data.
 2. The medicalguidance system (100) of claim 1, further comprising a computer system700 containing said medical equipment interface, wherein the medicalequipment system (200) comprises an ultrasound system coupled to thecomputer system via the medical equipment interface.
 3. The medicalguidance system (100) of claim 2, wherein the 3DGS (400) senses theuser's real-time movement of an ultrasound probe (215) relative to abody of a patient, and the stored reference outcome data comprisesultrasound images.
 4. The medical guidance system (100) of claim 1further comprising a computer system (700) containing said medicalequipment interface to the medical equipment system (200), wherein saidcomputer system is coupled to one or more of the ARUI (300), the 3DGS(400), the library (500), and the MLM (600).
 5. The medical guidancesystem of claim 1, wherein at least one of the fifth module and thesixth module comprises a neural network.
 6. The medical guidance systemof claim 5, wherein the neural network is a convolutional neuralnetwork.
 7. The medical guidance system of claim 1, wherein the 3DGS(400) comprises one or more of a magnetic GPS system, a digital cameratracking system, an infrared camera system, an accelerometer, and agyroscope.
 8. The medical device system of claim 1, wherein the 3DGS(400) senses real-time user positioning data by sensing at least one of:a magnetic field generated by said at least a portion of the medicalequipment system; the movement of one or more passive visual markerscoupled to one or more of the patient, a hand of the user, or a portionof the medical equipment system; and the movement of one or more activevisual markers coupled to one or more of the patient, a hand of theuser, or a portion of the medical equipment system.
 9. The medicalguidance system of claim 1, wherein the library (500) further includesone or more of: instructions for performing one or more medicalprocedures using the medical equipment system (200); generic dataassociated with the use of the medical equipment system on a number ofdifferent patients; and patient-specific data relating to the use of themedical equipment system on one or more specific patients.
 10. Themedical guidance system of claim 1, wherein at least one of saidreal-time position-based 3D AR feedback and said real-time outcome-based3D AR feedback is selected from a virtual still image, a virtual videoimage, sounds, or tactile information.
 11. The medical guidance systemof claim 1, wherein at least one of said real-time position-based 3D ARfeedback and said real-time outcome-based 3D AR feedback is selectedfrom: a virtual prompt indicating a movement correction to be performedby a user; a virtual image or video instructing the user to change theorientation of a probe to match a desired orientation; a virtual imageor video of a correct motion path to be taken by the user in performinga medical procedure; a color-coded image or video indicating correct andincorrect portions of the user's motion in performing a medicalprocedure; an instruction to a user to press an ultrasound probe deeperor shallower into tissue to focus the ultrasound image on a desiredtarget structure of the patient's body; and an auditory instruction,virtual image, or virtual video indicating a direction for the user tomove an ultrasound probe.
 12. The medical guidance system of claim 1,wherein the MLM 600 provides a first 3D AR feedback to the usercomprising one of said real-time position-based 3D AR feedback and saidreal-time outcome-based 3D AR feedback, and a second 3D AR feedback tothe user comprising the other of said real-time position-based 3D ARfeedback and said real-time outcome-based 3D AR feedback.
 13. Themedical guidance system (100) of claim 1 further comprising: a computersystem (700) containing said interface to the medical equipment system(200), and wherein said ARUI (300) comprises a software interfacelocated in said computer (700); and a head mounted display (HMD) coupledto said ARUI.
 14. The medical guidance system (100) of claim 1 furthercomprising: a computer system (700), wherein said ARUI (300) comprises asoftware interface located in said computer (700) and a head mounteddisplay (HMD) coupled to said software interface.
 15. The medicalguidance system (100) of claim 1, wherein said third module providessaid real-time position-based 3D AR feedback to the user visually in ahead mounted display relative to at least a portion of the patient'sbody, and said sixth module provides said real-time outcome-based 3D ARfeedback to the user visually in a head mounted display relative to atleast a portion of the patient's body.
 16. The medical guidance systemof claim 1, wherein the three-dimensional guidance system (3DGS) (400)is capable of sensing real-time user positioning data relating to atleast a portion of the medical equipment system (200) within a volume ofthe user's environment during said medical procedure performed by theuser
 17. A medical guidance system (100) for providing real-time,three-dimensional (3D) augmented reality (AR) feedback guidance in theuse of a medical equipment system (200), the medical guidance systemcomprising: a computer 700 comprising a medical equipment interface to amedical equipment system (200), wherein said medical equipment interfacereceives data from the medical equipment system during a medicalprocedure performed by a user to achieve a medical procedure outcome; anAR interface to an AR head mounted display (HMD) for presentinginformation pertaining to both real and virtual objects to the userduring the performance of the medical procedure; a guidance systeminterface (GSI) to a three-dimensional guidance system (3DGS) (400) thatsenses real-time user positioning data relating to one or more of themovement, position, and orientation of at least a portion of the medicalequipment system (200) within a volume of a user's environment during amedical procedure performed by the user; a library (500) containing 1)stored reference positioning data relating to one or more of themovement, position, and orientation of at least a portion of the medicalequipment system (200) during a reference medical procedure and 2)stored reference outcome data relating to an outcome of a referenceperformance of the reference medical procedure; and a machine learningmodule (MLM) (600) for providing at least one of 1) position-based 3D ARfeedback to the user based on the sensed user positioning data and 2)outcome-based 3D AR feedback to the user based on the medical procedureoutcome, the MLM (600) comprising a position-based feedback modulecomprising a first module for receiving and analyzing real-time userpositioning data; a second module for comparing the user positioningdata to the stored reference positioning data, and a third module forgenerating real-time position-based 3D AR feedback based on the outputof the second module, and providing said real-time position-based 3D ARfeedback to the user via the ARUI (300); and an outcome-based feedbackmodule comprising a fourth module for receiving real-time data from themedical equipment system (200) via said medical equipment interface asthe user performs the medical procedure; a fifth module for comparingthe real-time data received from the medical equipment system (200) asthe user performs the medical procedure to the stored reference outcomedata, and a sixth module for generating real-time outcome-based 3D ARfeedback based on the output of the fifth module, and providing saidreal-time outcome-based 3D AR feedback to the user via the ARUI (300).18. The medical guidance system (100) of claim 16, further comprisingthe AR HMD.